Two-Stage Nested Array Direction of Arrival Estimation for Mixed Signals

In this paper, a novel two-stage subspace-based direction of arrival (DOA) estimation algorithm with the nested array is proposed for mixed signals containing circular and non-circular ones. By exploiting the difference between the two types of steering vectors, the DOAs of circular signals are estimated in the first stage. After eliminating the estimated information of circular signals by the covariance matrix reconstruction and oblique projection methods, the dimensions of the noise subspace are increased in estimating the DOAs of non-circular signals in the second stage. Through the two-stage estimation, the DOAs of the circular and non-circular signals are estimated separately and different types of signals with similar or the same DOAs can be distinguished. Furthermore, to avoid the two-dimensional (2-D) search with huge computational burden, a one-dimensional (1-D) search method exploiting the rank deficiency is proposed in the DOA estimation for non-circular signals. The simulation results show that the proposed algorithm can effectively improve the estimation accuracy and resolution probability, especially when circular and non-circular signals have similar DOAs.


Introduction
DOA estimation is a key problem in array signal processing and has been widely used in communication, radar, detection, covert transmission, and other fields [1]. A source localization method based on multiple signal classification (MUSIC) was proposed in [2]. The DOAs were calculated by the estimation of signal parameters via the rotational invariant techniques (ESPRIT) in [3], where a novel ESPRIT algorithm based on geometric algebra (GA-ESPRIT) was proposed and the DOA was estimated with double parallel uniform linear arrays (ULA). A fast Fourier transform (FFT)-based method was proposed in [4] with low computational burden. A MUSIC-like method for non-circular signals (NC-MUSIC) was presented in [5], where a virtual array with twice the number of physical array sensors was constructed.
In recent years, the non-uniform linear array (NLA) has been applied to DOA estimation. A compressed sparse array (CSA) was proposed in [6] by combining the compressive sensing and the sparse array. An off-grid DOA estimation method was presented in [7], where a mismatch between true DOA and discrete angular grid was resolved. The estimation performance was improved with NLA and fourth-order cumulants in [8,9]. Specifically, a novel array structure was constructed with the co-prime array in [8], and DOAs was estimated via the sparse Bayesian learning. An array model was proposed in [9], where the fourth-order cumulants and the sparse reconstruction algorithm were exploited to estimate the DOA. An enhanced nested array (ENA) structure was proposed in [10] and a compressed sensing method was applied to DOA estimation. For mixed signals scenarios, the sparse reconstruction and root-MUSIC were used in [11] for non-circular signals and circular signals, respectively. The computational burden of these sparse reconstruction

Mixed Circular and Non-Circular Signals Model
Any digital modulation signal in the complex plane can be expressed as [22] where ϕ denotes the rotation phase, k controls the signal amplitude, s I (t) denotes the co-directional component, and s Q (t) denotes the orthogonal component. E{|s I (t)| 2 } = 1, E s Q (t) 2 = 1 and E s I (t)s Q (t) = 0.
To measure the degree of non-circularity, the definition of a signal's non-circular rate ρ is given by [23] where |ρ|= 0 and 0 <|ρ|≤ 1 represent the circular and non-circular signals, respectively, and |ρ|= 1 indicates the strictly non-circular signal. For convenience, only the case of mixed strictly non-circular and circular signals is considered in this paper. Then the circular signal can be represented as s c (t) = e −jϕ s IQ (t), and the non-circular signal can be represented as s n (t) = e −jϕ s I I (t), where s IQ (t) = σ 1 2 s I (t) + j 1 2 s Q (t) and s I I (t) = σs I (t). For the rest of this paper, the assumptions are required as follows: (1) All signals are statistically independent, zero-mean uncorrelated narrowband stationary processes. (2) Every sensor has zero-mean, additive white Gaussian noise and the noise is independent from the signals.

The Nested Array, cDCA, and cSCA Model
Consider a nested array having N 1 elements with inter-element spacing d and N 2 elements with inter-element spacing (N 1 + 1)d, where d is λ 2 in this paper and λ denotes the wavelength of the carrier wave. Assume L to be a set of integers, defined as Then the sensor locations of the nested array can be described as L = dL = {l 1 , l 2 , · · · , l N } (4) where N = N 1 + N 2 is the total number of array elements. Figure 1 shows the model of the nested array, where N 1 = N 2 = 4.  Assume M uncorrelated narrowband signals impinge on a nested array. Then, the received signals on the array can be expressed as  (5) where x(t) and n(t) are ' 1 N  vectors, and Assume M uncorrelated narrowband signals impinge on a nested array. Then, the received signals on the array can be expressed as where x(t) and n(t) are N × 1 vectors, and a(θ m ) = [e −j2πl 1 sinθ m /λ , e −j2πl 2 sinθ m /λ , . . . , e −j2πl N sinθ m /λ ] T denotes the m-th steering vector and M 1 indicates the number of non-circular signals. Figures 2 and 3 show the locations and number of elements in the cDCA and cSCA, respectively. According to [17], the total number of elements in the cDCA is 2R 1 + 1, and the number of elements in the cSCA is 2( The distance between the adjacent virtual elements is d. respectively. According to [17], the total number of elements in the cDCA is the number of elements in the cSCA is The distance between the adjacent virtual elements is d.    respectively. According to [17], the total number of elements in the cDCA is 1

21
R + , and the number of elements in the cSCA is The distance between the adjacent virtual elements is d.

The Proposed Method
Define the matrix J as Then, the covariance matrix of J can be expressed as For convenience, R J is further rewritten as where and σ 2 n represents the noise power.

Non-Circular Signals
For non-circular signals, only E where σ 2 m represents the power of the m-th non-circular signal and Φ N = diag{e −jϕ 1 , e −jϕ 2 , · · · , e −jϕ M 1 } is a diagonal matrix which contains the rotation phases information of the non-circular signals. We define the merged direction matrix A v and its m-th steering vector as Then E x N (t)x T N (t) can be rewritten as And According to [17], after vectorizing and removing the redundant elements in (18), (22) and (23), the equivalent signals on the cDCA and cSCA can be expressed as y δ = C δ p n (26)

Circular Signals
According to the definition of circular signals, (16). Then, the information of circular signals only exists in E x C (t)x H C (t) and the cDCA parts of the steering vectors of circular signals have the same structure as those of non-circular signals. The circular signals on the cDCA are sents the power vector of the circular signals.

DOA Estimation of Mixed Signals
According to the noise assumption, noise only exists on the cDCA. Then, the constructed signals and noise on the cDCA can be expressed as n d is a column vector in which only the R 1 + 1 -th element is 1 and the other elements are 0.
The constructed signals on the cSCA can be expressed as where ν m (M 1 + 1 ≤ m ≤ M) and τ m (M 1 + 1 ≤ m ≤ M) are all zero vectors. So far, the structures of the cDCA and cSCA have been derived. The equivalent received signal vector p can be regarded as the signals of a single snapshot, which needs to be smoothed to increase the rank of the covariance matrix in our method. Here, the method of smoothing the signals on the cDCA, the positive and negative cSCA separately is just like that in [17].
First, the signals on the cDCA are smoothed. In the cDCA, the steering vectors of circular signals and non-circular signals have the same form, then D e can be expressed as Next, the signals on the positive and negative cSCA are smoothed separately. To ensure that the smoothed matrix has the same number of columns as Y e , the positive and negative cSCA are also divided into R 1 + 1 subarrays, and each subarray contains R 2 − R 1 + 1 elements. Then, the constructed equivalent signals can be expressed as where m = [e −jR 1 π sin θ m e −j2ϕ m , e −j(R 1 +1)π sin θ m e −j2ϕ m , · · · , e −jR 2 π sin θ m e −j2ϕ m ] By arranging the equivalent signals on the R 1 + 1 subarrays, the spatially smoothed signal matrices Y β and Y σ can be rewritten as and Y σ are concatenated as (52) to form a new received signal matrix Y.
After these steps, an uneven nested array can be transformed into a ULA, and the number of equivalent elements is significantly increased.
The covariance matrix of Y can be calculated as The eigenvalues decomposition of R χ can be expressed as is a diagonal matrix in which the diagonal elements are the 2R 2 − R 1 + 3 − M small eigenvalues, and their corresponding eigenvector matrix U nc constitutes the noise subspace.
It can be seen from (54) and (55) that the cSCA parts of the steering vectors of the circular signals are zero vectors while those of the non-circular signals contain the angles and rotation phases information. By exploiting the difference between two types of steering vectors, we can distinguish the two types of signals and estimate their DOAs separately.

DOAs of Circular Signals
By exploiting the orthogonal property of the signal subspace and noise subspace, the space spectrum P 1 can be expressed as After the one-dimensional spectral peak search, the angles corresponding to the maximum M − M 1 peaks are the DOAs of circular signals. Due to the differences in steering vectors of different types of signals, only the DOAs of circular signals can be estimated in the peak search of (58). Since the cDCA part is contained in the steering vectors of both non-circular and circular signals, the information of circular signals on the cDCA needs to be eliminated after the DOAs of circular signals are estimated.

Eliminate the Information of Circular Signals
Two methods are proposed to eliminate the estimated circular signals based on the total number of signals.

•
Reconstruct the covariance matrix When the total number of circular and non-circular signals does not exceed N , the power of the circular signals can be calculated from the covariance matrix E x(t)x H (t) , and the information of circular signals can be eliminated like the method in [24].
From the previous derivation, the information of the circular signals only exists in E x(t)x H (t) . Let R h be the covariance matrix without noise. From the DOAs estimation of circular signals, the power can be calculated as Then the covariance matrix E x N (t)x H N (t) containing only DOAs of non-circular signals is obtained as Then, the redundant elements in E x N (t)x H N (t) are removed. According to the smoothing method in (38)-(41), we have T represents the power vector of the non-circular signals.
The equivalent signals on the cSCA do not contain the information of circular signals and noise, therefore, it is not necessary to smooth the signals on the cSCA again when calculating the DOAs of non-circular signals.
By splicing Y σ , Y ε and Y β together, we can obtain the equivalent received non-circular signals matrix after removing noise and circular signals information.
The covariance matrix of the non-circular signals can be calculated as The above method is no longer applicable when the number of signals exceeds N . Another method is proposed by exploiting the oblique projection method. The proof of the oblique projection method is presented in the Appendix A. The oblique projection operator is constructed as where A SC (θ) donates the equivalent steering vector matrix of the smoothed circular signals in (53). Then, the covariance matrix is obtained as It should be noted that the oblique projection algorithm is also applicable when the total number of signals is no more than N .

DOAs of Non-Circular Signals
Through the above derivation, the two methods for eliminating circular signals are proposed, and the covariance matrix R ν of non-circular signals is obtained. By exploiting the eigenvalue decomposition, R ν can be expressed as The eigenvectors corresponding to the smaller 2R 2 − R 1 + 3 − M 1 eigenvalues represent the noise subspace. According to the subspace theory, the signal subspace and noise subspace are orthogonal.
After performing a 2-D spectral peak search on (69), the DOAs and rotation phases of non-circular signals can be obtained. To reduce the amount of calculation, a method converting the 2-D search into 1-D search is proposed.
a ε (θ m , ϕ m ) is rewritten as the form of the multiplication of two matrices in (70).
It can be seen from (70) that Q only contains the angle information of the signals, and K is considered to be a coefficient matrix and Ξ is the solution vector. Here, the problem can be transformed into the homogeneous equation KΞ= O existing at least one non-zero solution Ξ =   e j2ϕ m 1 e −j2ϕ m   , which illustrates that the rank of the coefficient matrix K is less than the number of unknowns. Therefore, the rank estimator method can be used to estimate the DOAs of non-circular signals.
Since all DOAs are in [−90 • , 90 • ], they can be obtained by searching θ. Because the rank of K changes with different θ, the correct DOAs can obtained when rank of K is deficient. The rotational phases can be further estimated from the singular vectors of K in our method, but they are not considered in many papers [5,14,17] and also in our simulations.
If the traditional 2-D search is applied to (69), it can be summarized as: where ∆φ and ∆θ denote the search interval of the rotation phase and DOA, respectively. The proposed 1-D search method in this paper avoids the 2-D search and the computational burden is significantly reduced. The angle search process of the proposed method is summarized as: It can be seen that the proposed method only needs to perform 1-D search for one time in estimating DOAs of non-circular signals and the computational burden is low.
In summary, the main steps of the proposed method can be described as follows: (1) Construct matrix J via (13) and calculate covariance matrix R J via (14).
(2) Construct y d , y δ , and y φ from the covariance matrix and exploit spatial smoothing. i. The covariance matrix of the reconstructed circular signals is eliminated by (60). After the related elements being extracted and smoothed, the covariance matrix of non-circular signals can be obtained as (65). ii.
Calculate the oblique projection operator based on the estimated DOAs of the circular signals in (66). Then, the covariance matrix of the non-circular signals can be obtained by (67).
(5) Perform eigenvalue decomposition on the covariance matrix to obtain the noise subspace. The steering vector of the non-circular signal is decomposed as (70). Then, construct matrix K in (72) and the DOAs of non-circular signals are determined by searching for the rank deficiency of K.

Discussion A: Computational Complexity
For convenience, we define ∆θ as the angle search interval and F as the number of virtual array elements. M 2 = M − M 1 is the number of circular signals. The computational complexity of [17] mainly comes from calculating the inverse of the matrix for both circular and non-circular signals. The overall complexity of this method is . The complexity of the proposed algorithm mainly comes from the removal of circular signals and the singular value decomposition. If the covariance matrix reconstruction method is used, the complexity ). By estimating different types of signals separately, the complexity is increased, but the accuracy of estimation is also improved. When the different types of signals are estimated simultaneously in our method, the complexity mainly comes from singular value decomposition, which is similar to the complexity in [17].

Discussion B: The Scenario of Unmixed Signals
The proposed method can also work well for unmixed signals. For circular signals, only cDCA part and one parameter θ exist in this problem, and the traditional MUSIC algorithm can be exploited to estimate the DOA directly through the 1-D spectral peak search. For non-circular signals, the problem in (70) can be transformed into the homogeneous equation existing at least one non-zero solution and the rank of the coefficient matrix K is less than three, which has been discussed in Section 3.3.3. To sum up, the proposed method can be applied to the scenario of unmixed signals, and only the one-stage estimation is needed.

Simulation Results
In this section, simulation results are presented to verify the performance of our algorithm where the condition number method is chosen as the indication of the rank deficiency in the estimation of non-circular signals. In this paper, the resolution probability and RMSE are considered. In our simulations, all signals have the same power, and the signal-to-noise ratio (SNR) is defined as where s 2 i is the power of the i-th signal and n 2 j is the noise power of the j-th sensor.

Comparison of Resolution Probability
The resolution probability when circular and non-circular signals have similar DOAs is analyzed first. In this simulation, when the angle error is less than , the estimation is considered to be successful [17], where θ c denotes the DOA of the circular signal and θ n denotes the DOA of the non-circular one. Figure 4 shows the resolution probability versus the number of snapshots when SNR is −14 dB, where θ c = 20 • , θ n = 22 • , the non-circular signal phase ϕ= 5 • , and N 1 = N 2 = 4. From simulation results of 500 Monte Carlo experiments, the proposed method has the highest resolution probability and obvious advantages compared with other methods.  When the SNR is 0 dB and the number of snapshots is 200, the resolution probability of different angle intervals is simulated. As shown in Figure 5, when the angle interval is 0.5 , the resolution probability of the proposed method is more than 0.4, which is the highest among all methods. With the increase of the angle interval, the proposed method maintains the highest resolution probability. When the SNR is 0 dB and the number of snapshots is 200, the resolution probability of different angle intervals is simulated. As shown in Figure 5, when the angle interval is 0.5 • , the resolution probability of the proposed method is more than 0.4, which is the highest among all methods. With the increase of the angle interval, the proposed method maintains the highest resolution probability.
From the above simulations, the proposed method has the highest resolution probability, and the advantage is more obvious with few snapshots and low SNR. Unlike other methods where all signals are estimated simultaneously, the separate estimation of different types of signals in our method increases the dimensions of the noise subspace in calculating the DOAs of non-circular signals and the two types of signals with similar DOAs can be distinguished. Our method achieves the highest resolution probability through the two-stage estimation.
When the SNR is 0 dB and the number of snapshots is 200, the resolution probability of different angle intervals is simulated. As shown in Figure 5, when the angle interval is 0.5 , the resolution probability of the proposed method is more than 0.4, which is the highest among all methods. With the increase of the angle interval, the proposed method maintains the highest resolution probability. From the above simulations, the proposed method has the highest resolution probability, and the advantage is more obvious with few snapshots and low SNR. Unlike other methods where all signals are estimated simultaneously, the separate estimation of different types of signals in our method increases the dimensions of the noise subspace in calculating the DOAs of non-circular signals and the two types of signals with similar DOAs can be distinguished. Our method achieves the highest resolution probability through the two-stage estimation.
Next, the resolution probability is simulated when the same types of signals impinge on the array. The resolution probabilities of different methods by estimating the two non-  Figure 6 also shows that our method has the best performance, and the highest resolution probability of the proposed method reflects the advantages of searching for the maximum condition number method.

Comparison of RMSE
In this section, the RMSEs of different methods are compared to analyze the accuracy of the proposed algorithm.  Table 1 and 12 4 NN == .  Figure 6 also shows that our method has the best performance, and the highest resolution probability of the proposed method reflects the advantages of searching for the maximum condition number method.

Comparison of RMSE
In this section, the RMSEs of different methods are compared to analyze the accuracy of the proposed algorithm.
where V is the simulation times, M is the number of signals,θ m,v is the estimated value of the m-th signal angle in the v-th experiment, and θ m is the theoretical value of the m-th angle. The performance of RMSE versus SNR and the snapshot number is simulated through 500 Monte Carlo experiments, where the DOAs and rotation phases of sources are shown in Table 1 and N 1 = N 2 = 4. When the number of snapshots is 1000, the RMSE is shown in Figure 7. When the SNR is 0 dB, Figure 8 shows the curve of RMSE versus the number of snapshots. It can be indicated that the proposed method has the best performance.  Unlike the SS-MUSIC method in [14], where only the cDCA is used, the cDCA and cSCA are exploited simultaneously in our method. Furthermore, the circular signals are eliminated before calculating the DOAs of non-circular ones in our method and the RMSE is smaller than estimating all signals together in [14,17]. The small number of snapshots in [18] (QS-MUSIC) leads to poor accuracy. The RMSE of the methods in [2,5] are more than 10 . Thus, the proposed method has the best performance.
When only a circular and a non-circular signal have similar DOAs, the proposed method has the minimum RMSE. If there are many signals in different types with similar or the same DOAs, the advantages of our method will be more obvious.
Next, the RMSE is verified in Figures 9 and 10 when the DOAs of circular signals and non-circular signals are distributed uniformly. In this simulation, the source information  Unlike the SS-MUSIC method in [14], where only the cDCA is used, the cDCA and cSCA are exploited simultaneously in our method. Furthermore, the circular signals are eliminated before calculating the DOAs of non-circular ones in our method and the RMSE is smaller than estimating all signals together in [14,17]. The small number of snapshots in [18] (QS-MUSIC) leads to poor accuracy. The RMSE of the methods in [2,5] are more than 10 . Thus, the proposed method has the best performance.
When only a circular and a non-circular signal have similar DOAs, the proposed method has the minimum RMSE. If there are many signals in different types with similar or the same DOAs, the advantages of our method will be more obvious.
Next, the RMSE is verified in Figures 9 and 10 when the DOAs of circular signals and non-circular signals are distributed uniformly. In this simulation, the source information 4 NN == Unlike the SS-MUSIC method in [14], where only the cDCA is used, the cDCA and cSCA are exploited simultaneously in our method. Furthermore, the circular signals are eliminated before calculating the DOAs of non-circular ones in our method and the RMSE is smaller than estimating all signals together in [14,17]. The small number of snapshots in [18] (QS-MUSIC) leads to poor accuracy. The RMSE of the methods in [2,5] are more than 10 • . Thus, the proposed method has the best performance.
When only a circular and a non-circular signal have similar DOAs, the proposed method has the minimum RMSE. If there are many signals in different types with similar or the same DOAs, the advantages of our method will be more obvious.
Next, the RMSE is verified in Figures 9 and 10 when the DOAs of circular signals and non-circular signals are distributed uniformly. In this simulation, the source information is shown in Table 2 and N 1 = N 2 = 4. Figure 9 shows the performance of RMSE versus SNR for different methods when the number of snapshots is 800. Figure 10 shows the RMSE when the number of snapshots is changed from 400 to 1200 under the condition of SNR = 0 dB and the other conditions remain unchanged. The RMSE of three methods decreases with the increase of the number of snapshots and our method has the best performance. The dimensions of the noise subspace reach the maximum for non-circular signals in our method, which can improve the accuracy of estimation.
When the angle interval of different types of signals becomes larger, the RMSEs of the different methods are close, because the interference between adjacent impinging signals has been reduced. Of course, in this case, the one-stage angle search without eliminating circular signals can also be exploited in our method with low computational burden.  The relationship between the RMSE and the number of array elements is simulated, and three different cases are compared. The sources information is shown in Table 2. Figure 11 shows the change of RMSE with SNR, and the snapshot number is fixed at 800. It can be seen from the simulation results that when SNR is fixed, the RMSE decreases with the increase of array elements.  Figure 9 shows the performance of RMSE versus SNR for different methods when the number of snapshots is 800. Figure 10 shows the RMSE when the number of snapshots is changed from 400 to 1200 under the condition of SNR = 0 dB and the other conditions remain unchanged. The RMSE of three methods decreases with the increase of the number of snapshots and our method has the best performance. The dimensions of the noise subspace reach the maximum for non-circular signals in our method, which can improve the accuracy of estimation.
When the angle interval of different types of signals becomes larger, the RMSEs of the different methods are close, because the interference between adjacent impinging signals has been reduced. Of course, in this case, the one-stage angle search without eliminating circular signals can also be exploited in our method with low computational burden.  The relationship between the RMSE and the number of array elements is simulated, and three different cases are compared. The sources information is shown in Table 2. Figure 11 shows the change of RMSE with SNR, and the snapshot number is fixed at 800. It can be seen from the simulation results that when SNR is fixed, the RMSE decreases with the increase of array elements.   Figure 9 shows the performance of RMSE versus SNR for different methods when the number of snapshots is 800. Figure 10 shows the RMSE when the number of snapshots is changed from 400 to 1200 under the condition of SNR = 0 dB and the other conditions remain unchanged. The RMSE of three methods decreases with the increase of the number of snapshots and our method has the best performance. The dimensions of the noise subspace reach the maximum for non-circular signals in our method, which can improve the accuracy of estimation.
When the angle interval of different types of signals becomes larger, the RMSEs of the different methods are close, because the interference between adjacent impinging signals has been reduced. Of course, in this case, the one-stage angle search without eliminating circular signals can also be exploited in our method with low computational burden.
The relationship between the RMSE and the number of array elements is simulated, and three different cases are compared. The sources information is shown in Table 2. Figure 11 shows the change of RMSE with SNR, and the snapshot number is fixed at 800. It can be seen from the simulation results that when SNR is fixed, the RMSE decreases with the increase of array elements. To test the performance of the method when the number of signals is more than the actual number of elements of the nested array, further simulations are carried out. When  The RMSE of the proposed method is always the smallest and can achieve better accuracy when the number of snapshots and SNR are increased, as all signals can be well distinguished by the two-stage estimation method and the dimensions of the noise subspace are increased by eliminating the circular signals first. In [14,17], the large error makes it unable to distinguish multiple spectral peaks when there are many signals. Even if SNR or the number of snapshots is increased, they cannot obtain the correct estimation of DOAs.
We also analyze the spatial spectrum of the proposed method when a circular signal and a non-circular signal have similar DOAs in Figure 13, where The SNR is −12 dB, the number of snapshots is 500 and 12 4 NN == .  Figure 12 shows the RMSE of the three methods and N 1 = N 2 = 4. To test the performance of the method when the number of signals is more than the actual number of elements of the nested array, further simulations are carried out. When  The RMSE of the proposed method is always the smallest and can achieve better accuracy when the number of snapshots and SNR are increased, as all signals can be well distinguished by the two-stage estimation method and the dimensions of the noise subspace are increased by eliminating the circular signals first. In [14,17], the large error makes it unable to distinguish multiple spectral peaks when there are many signals. Even if SNR or the number of snapshots is increased, they cannot obtain the correct estimation of DOAs.
We also analyze the spatial spectrum of the proposed method when a circular signal and a non-circular signal have similar DOAs in Figure 13, where The SNR is −12 dB, the number of snapshots is 500 and 12 4 NN == . The RMSE of the proposed method is always the smallest and can achieve better accuracy when the number of snapshots and SNR are increased, as all signals can be well distinguished by the two-stage estimation method and the dimensions of the noise subspace are increased by eliminating the circular signals first. In [14,17], the large error makes it unable to distinguish multiple spectral peaks when there are many signals. Even if SNR or the number of snapshots is increased, they cannot obtain the correct estimation of DOAs.
We also analyze the spatial spectrum of the proposed method when a circular signal and a non-circular signal have similar DOAs in Figure 13, where θ c = 22 • and θ n = 20 • . The SNR is −12 dB, the number of snapshots is 500 and N 1 = N 2 = 4. The DOA estimation results of circular and non-circular signals in the proposed method are marked on Figure 13. Unlike other related methods, our method can always distinguish between two spectral peaks with similar DOAs and has high accuracy.
The above methods are comprehensively compared in Table 3 and the superiority of the proposed method is illustrated. The rank deficiency is exploited in calculating the DOAs of non-circular signals; therefore, some different search methods can be evolved. Three representative methods are simulated in Figure 14. Circular and non-circular signals are estimated simultaneously.
Method 1: Find the peak value of the ratio of the maximum singular value to the minimum singular value (the condition number). Method 2: Find the peak value of the ratio of the sub small singular value to the minimum singular value. Method 3: Find the peak value of the reciprocal of the minimum singular value. The variation of RMSE with SNR is analyzed in Figure 14, where the number of snapshots is 800, 1 2 4 NN == and the source information is shown in Table 4. When different non-circular signals have similar DOAs, method 1 and method 2 both have high accuracy. Thus, the method of searching for the maximum condition number is selected to calculate the DOAs in this paper.  The DOA estimation results of circular and non-circular signals in the proposed method are marked on Figure 13. Unlike other related methods, our method can always distinguish between two spectral peaks with similar DOAs and has high accuracy.
The above methods are comprehensively compared in Table 3 and the superiority of the proposed method is illustrated. The rank deficiency is exploited in calculating the DOAs of non-circular signals; therefore, some different search methods can be evolved. Three representative methods are simulated in Figure 14. Circular and non-circular signals are estimated simultaneously.

Conclusions
In this paper, an algorithm is proposed to locate circular and non-circular signals separately with the nested array by exploiting different steering vectors of two types of signals. Only the DOAs of circular signals are estimated in the first stage after the con-  Method 2: Find the peak value of the ratio of the sub small singular value to the minimum singular value. Method 3: Find the peak value of the reciprocal of the minimum singular value. The variation of RMSE with SNR is analyzed in Figure 14, where the number of snapshots is 800, N 1 = N 2 = 4 and the source information is shown in Table 4. When different non-circular signals have similar DOAs, method 1 and method 2 both have high accuracy. Thus, the method of searching for the maximum condition number is selected to calculate the DOAs in this paper.

Conclusions
In this paper, an algorithm is proposed to locate circular and non-circular signals separately with the nested array by exploiting different steering vectors of two types of signals. Only the DOAs of circular signals are estimated in the first stage after the construction of equivalent received signals on the cDCA and cSCA. After eliminating the information of circular signals, the dimensions of noise subspace are increased in estimating DOAs of non-circular signals in the second stage and signals with two types are distinguished. By exploiting the rank deficiency, three ways are proposed to estimate the DOAs of non-circular signals and the method of searching for the maximum condition number is chosen, where the computational burden is significantly reduced by converting 2-D search to 1-D search. Compared with some other methods such as SD-RD-MUSIC and SS-MUSIC, the proposed method has the highest resolution probability and the best estimation accuracy because of the two-stage estimation, especially when two types of sources have similar or the same DOAs. Numerical simulation results demonstrate that the proposed method is more effective and has better performance than other ones.

Conflicts of Interest:
The authors declare no conflict of interest.

SN
where S DC is the circular signals part and S DN is the non-circular signals part. It can be seen from (A1) that to calculate the oblique projection operator E A SC (θ)A SN (θ) . It is necessary to know the DOAs of all sources, but they cannot be obtained in the actual environment, so the following formula is used as the approximate expression of E A SC (θ)A SN (θ) .
Then the covariance matrix can be expressed as